Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] benchmarks with dogstatsd #641

Closed
wants to merge 39 commits into from
Closed

[WIP] benchmarks with dogstatsd #641

wants to merge 39 commits into from

Conversation

VianneyRuhlmann
Copy link
Contributor

What does this PR do?

Attempt to stabilize benchmarks

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

ajgajg1134 and others added 30 commits September 9, 2024 16:14
@ajgajg1134
move dogstatsd-client to separate crate so it can be shared
75ecb44
@ajgajg1134
adding some initial stats and tests
ba57b1d
@ajgajg1134
make traceexporter mut
581c243
@ajgajg1134
Merge branch 'main' into andrew.glaude/dogstatsd-client
5746b4a
@ajgajg1134
miri can ignore unit test
7411f46
@ajgajg1134
Merge branch 'main' into andrew.glaude/dogstatsd-client
fe3cc6d
@ajgajg1134
refactor dogstatsd-client to not need mutability
6515250
@ajgajg1134
use vec default to avoid unneeded allocation
a388c80
@ajgajg1134
fmt
dfb1e51
@ajgajg1134
Merge branch 'main' into andrew.glaude/dogstatsd-client
77bfbd2
@ajgajg1134
add dogstatsd-client to docker build tool
062a891
@ajgajg1134
clean up cargo.toml
d9d21ee
@ajgajg1134
remove unneeded dep
61d5f83
@ajgajg1134
use AsRef for metric strings, improve perf and simplify usage
0bb2986
@ajgajg1134
simplify emitting stats in data-pipeline
2961334
@ajgajg1134
fix lint
b0b0256
@ajgajg1134
Remove copy of tags and pass to dogstatsd client by reference
2f65946
@ajgajg1134
Update licenses
4d25dc2
@ajgajg1134
Allow IntoIterator tags to support static and dynamic tags: WIP
37fd00e
@bwoebi
Fix lifetimes of DogstatsDAction
ad04e27 
Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
@bwoebi
Actually also allow &Vec<Tag> and &[Tag]
d30df06 
Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
@ajgajg1134
cargo fmt
242224c
@ajgajg1134
disable cargo bench in dogstatd-client, we use criterion
93f6cb6
@ajgajg1134
Merge branch 'main' into andrew.glaude/dogstatsd-client
5532b66
@ajgajg1134
Try using iterators in data-pipeline for metrics
8fecb9a
@ajgajg1134
bump ver
1141243
@ajgajg1134
more metrics
ab002f5
@ajgajg1134
Split dogstatsdaction type to simplify code
aef310a
@ajgajg1134
move to owned type for sidecar-ffi
1cb870a
@ajgajg1134
remove unused import
22b333d
@pr-commenter
Copy link

pr-commenter bot commented Sep 19, 2024 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2024-09-19 12:10:45

Comparing candidate commit 7f4735c in PR branch vianney/benchmarks-with-dogstatsd with baseline commit 2f1c45b in branch vianney/main-with-iterations.

Found 7 performance improvements and 24 performance regressions! Performance is the same for 20 metrics, 2 unstable metrics.

scenario:credit_card/is_card_number/

  • 🟥 execution_time [+120.153µs; +120.225µs] or [+99.981%; +100.040%]
  • 🟥 throughput [-416146731.916op/s; -416004826.948op/s] or [-50.011%; -49.994%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟥 execution_time [+516.242µs; +559.557µs] or [+4.639%; +5.028%]
  • 🟥 throughput [-429683.613op/s; -397277.253op/s] or [-4.782%; -4.421%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟥 execution_time [+400.646µs; +455.357µs] or [+3.853%; +4.379%]
  • 🟥 throughput [-402725.371op/s; -355012.005op/s] or [-4.188%; -3.691%]

scenario:credit_card/is_card_number/37828224631

  • 🟥 execution_time [+120.137µs; +120.165µs] or [+99.961%; +99.984%]
  • 🟥 throughput [-416021184.296op/s; -415924418.259op/s] or [-49.999%; -49.987%]

scenario:credit_card/is_card_number/378282246310005

  • 🟥 execution_time [+409.153µs; +458.864µs] or [+4.069%; +4.563%]
  • 🟥 throughput [-433906.708op/s; -387423.340op/s] or [-4.363%; -3.895%]

scenario:credit_card/is_card_number/37828224631000521389798

  • 🟥 execution_time [+835.689µs; +880.791µs] or [+8.740%; +9.212%]
  • 🟥 throughput [-881947.018op/s; -838840.446op/s] or [-8.432%; -8.020%]

scenario:credit_card/is_card_number_no_luhn/

  • 🟥 execution_time [+120.023µs; +120.067µs] or [+99.776%; +99.813%]
  • 🟥 throughput [-415359346.174op/s; -415094375.200op/s] or [-49.965%; -49.933%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

  • 🟩 execution_time [-371.977µs; -330.066µs] or [-3.976%; -3.528%]
  • 🟩 throughput [+390879.272op/s; +439530.841op/s] or [+3.656%; +4.111%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

  • 🟩 execution_time [-436.810µs; -400.049µs] or [-5.089%; -4.661%]
  • 🟩 throughput [+569567.241op/s; +619670.213op/s] or [+4.888%; +5.318%]

scenario:credit_card/is_card_number_no_luhn/37828224631

  • 🟥 execution_time [+120.150µs; +120.178µs] or [+99.970%; +99.993%]
  • 🟥 throughput [-416032814.593op/s; -415938126.403op/s] or [-50.001%; -49.990%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

  • 🟩 execution_time [-463.096µs; -428.167µs] or [-5.601%; -5.178%]
  • 🟩 throughput [+660833.541op/s; +712116.848op/s] or [+5.463%; +5.887%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

  • 🟥 execution_time [+728.675µs; +785.401µs] or [+7.561%; +8.150%]
  • 🟥 throughput [-784576.389op/s; -729113.354op/s] or [-7.559%; -7.025%]

scenario:normalization/normalize_name/normalize_name/good

  • 🟥 execution_time [+362.225ns; +372.929ns] or [+2.359%; +2.428%]
  • 🟥 throughput [-1543990.251op/s; -1500077.792op/s] or [-2.371%; -2.304%]

scenario:normalization/normalize_service/normalize_service/[empty string]

  • 🟥 execution_time [+1.223µs; +1.259µs] or [+2.800%; +2.882%]
  • 🟥 throughput [-641417.916op/s; -623196.678op/s] or [-2.802%; -2.723%]

scenario:normalization/normalize_service/normalize_service/test_ASCII

  • 🟥 execution_time [+1.510µs; +1.559µs] or [+3.002%; +3.101%]
  • 🟥 throughput [-597848.768op/s; -579239.471op/s] or [-3.007%; -2.913%]

scenario:sql/obfuscate_sql_string

  • 🟩 execution_time [-7.477µs; -7.412µs] or [-9.885%; -9.798%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 1.193µs 1.385µs ± 0.049µs 1.395µs ± 0.027µs 1.419µs 1.446µs 1.471µs 1.472µs 5.49% -1.226 2.788 3.55% 0.003µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [1.378µs; 1.392µs] or [-0.494%; +0.494%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 18.249µs 25.007µs ± 10.734µs 20.136µs ± 1.434µs 33.854µs 43.230µs 46.893µs 99.904µs 396.14% 3.138 15.413 42.82% 0.759µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [23.519µs; 26.494µs] or [-5.949%; +5.949%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 297.167ns 308.957ns ± 13.351ns 304.138ns ± 4.523ns 309.565ns 344.147ns 349.252ns 350.698ns 15.31% 1.736 2.059 4.31% 0.944ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [307.106ns; 310.807ns] or [-0.599%; +0.599%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 67.901µs 68.201µs ± 0.192µs 68.154µs ± 0.068µs 68.265µs 68.443µs 68.618µs 70.136µs 2.91% 5.407 49.851 0.28% 0.014µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [68.175µs; 68.228µs] or [-0.039%; +0.039%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 240.160µs 240.366µs ± 0.256µs 240.345µs ± 0.067µs 240.401µs 240.531µs 240.750µs 243.666µs 1.38% 10.861 136.276 0.11% 0.018µs 1 200
credit_card/is_card_number/ throughput 410397198.042op/s 416033442.219op/s ± 438509.210op/s 416068856.632op/s ± 115850.604op/s 416199103.311op/s 416266883.210op/s 416293915.938op/s 416389899.571op/s 0.08% -10.797 135.120 0.11% 31007.284op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 11.337ms 11.667ms ± 0.138ms 11.658ms ± 0.087ms 11.745ms 11.899ms 11.975ms 12.195ms 4.61% 0.439 0.401 1.18% 0.010ms 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 8199882.299op/s 8572674.201op/s ± 101262.483op/s 8578077.735op/s ± 63961.889op/s 8642144.527op/s 8730012.632op/s 8781896.458op/s 8820681.888op/s 2.83% -0.362 0.266 1.18% 7160.339op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 10.407ms 10.827ms ± 0.168ms 10.827ms ± 0.118ms 10.952ms 11.089ms 11.204ms 11.247ms 3.88% -0.070 -0.401 1.55% 0.012ms 1 200
credit_card/is_card_number/ 378282246310005 throughput 8891646.485op/s 9238223.828op/s ± 143352.482op/s 9236498.277op/s ± 100671.696op/s 9333895.963op/s 9493317.317op/s 9561284.589op/s 9609219.008op/s 4.04% 0.144 -0.388 1.55% 10136.551op/s 1 200
credit_card/is_card_number/37828224631 execution_time 240.204µs 240.335µs ± 0.089µs 240.329µs ± 0.069µs 240.389µs 240.501µs 240.609µs 240.682µs 0.15% 0.964 1.006 0.04% 0.006µs 1 200
credit_card/is_card_number/37828224631 throughput 415485295.233op/s 416086269.152op/s ± 153792.874op/s 416095546.635op/s ± 120061.475op/s 416219787.832op/s 416270713.928op/s 416301311.055op/s 416313337.242op/s 0.05% -0.962 0.998 0.04% 10874.798op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 10.100ms 10.490ms ± 0.146ms 10.485ms ± 0.101ms 10.589ms 10.714ms 10.814ms 10.839ms 3.37% -0.127 -0.172 1.39% 0.010ms 1 200
credit_card/is_card_number/378282246310005 throughput 9226197.863op/s 9535101.684op/s ± 133191.087op/s 9537246.629op/s ± 91253.152op/s 9625395.490op/s 9765382.235op/s 9886658.058op/s 9901260.668op/s 3.82% 0.202 -0.133 1.39% 9418.032op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 10.221ms 10.420ms ± 0.141ms 10.373ms ± 0.085ms 10.516ms 10.665ms 10.757ms 11.032ms 6.35% 1.025 0.862 1.35% 0.010ms 1 200
credit_card/is_card_number/37828224631000521389798 throughput 9064196.903op/s 9598620.039op/s ± 128416.720op/s 9640141.338op/s ± 78814.658op/s 9701746.254op/s 9743930.997op/s 9777627.241op/s 9783506.651op/s 1.49% -0.955 0.572 1.33% 9080.433op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 2.163ms 2.195ms ± 0.025ms 2.189ms ± 0.012ms 2.202ms 2.243ms 2.273ms 2.335ms 6.70% 1.766 5.055 1.14% 0.002ms 1 200
credit_card/is_card_number/x371413321323331 throughput 42821219.524op/s 45568030.900op/s ± 508892.639op/s 45692249.008op/s ± 253931.045op/s 45929077.421op/s 46180909.689op/s 46234526.927op/s 46238840.256op/s 1.20% -1.639 4.264 1.11% 35984.144op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 240.178µs 240.337µs ± 0.081µs 240.338µs ± 0.063µs 240.392µs 240.481µs 240.538µs 240.553µs 0.09% 0.393 -0.539 0.03% 0.006µs 1 200
credit_card/is_card_number_no_luhn/ throughput 415709215.759op/s 416081971.618op/s ± 140191.410op/s 416080603.930op/s ± 109797.233op/s 416201340.768op/s 416267634.082op/s 416330439.368op/s 416358487.037op/s 0.07% -0.391 -0.541 0.03% 9913.030op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 8.929ms 9.005ms ± 0.072ms 8.964ms ± 0.021ms 9.066ms 9.138ms 9.207ms 9.292ms 3.66% 1.272 1.075 0.80% 0.005ms 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 10762027.992op/s 11105803.917op/s ± 88035.525op/s 11155698.585op/s ± 25700.982op/s 11169514.331op/s 11186338.244op/s 11196480.152op/s 11199844.367op/s 0.40% -1.238 0.924 0.79% 6225.052op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 8.135ms 8.165ms ± 0.034ms 8.158ms ± 0.008ms 8.165ms 8.247ms 8.307ms 8.382ms 2.75% 3.252 12.621 0.41% 0.002ms 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 11930375.284op/s 12247094.520op/s ± 50059.054op/s 12258655.051op/s ± 12327.762op/s 12270994.671op/s 12284300.694op/s 12289385.534op/s 12291818.842op/s 0.27% -3.203 12.185 0.41% 3539.710op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 240.217µs 240.349µs ± 0.089µs 240.353µs ± 0.066µs 240.399µs 240.503µs 240.634µs 240.712µs 0.15% 0.929 1.122 0.04% 0.006µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 415434910.400op/s 416061131.555op/s ± 154153.302op/s 416054369.206op/s ± 113618.183op/s 416189483.546op/s 416251334.141op/s 416277297.484op/s 416289982.051op/s 0.06% -0.926 1.113 0.04% 10900.285op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 7.783ms 7.823ms ± 0.029ms 7.816ms ± 0.009ms 7.826ms 7.884ms 7.932ms 7.974ms 2.02% 2.635 8.367 0.37% 0.002ms 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 12541134.136op/s 12783268.023op/s ± 46480.854op/s 12794686.425op/s ± 14347.833op/s 12807149.673op/s 12824844.130op/s 12844668.584op/s 12848115.679op/s 0.42% -2.598 8.137 0.36% 3286.693op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 10.219ms 10.394ms ± 0.133ms 10.353ms ± 0.069ms 10.464ms 10.660ms 10.808ms 10.916ms 5.44% 1.452 2.186 1.27% 0.009ms 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 9160968.010op/s 9622625.096op/s ± 120513.297op/s 9658880.225op/s ± 65105.459op/s 9711518.148op/s 9752427.897op/s 9767722.214op/s 9785703.850op/s 1.31% -1.375 1.882 1.25% 8521.577op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 2.162ms 2.193ms ± 0.020ms 2.188ms ± 0.012ms 2.206ms 2.228ms 2.259ms 2.276ms 4.02% 1.102 1.720 0.92% 0.001ms 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 43936067.047op/s 45605107.782op/s ± 417078.878op/s 45700438.745op/s ± 258889.504op/s 45904138.997op/s 46181879.629op/s 46234894.385op/s 46251402.458op/s 1.21% -1.035 1.468 0.91% 29491.930op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [240.330µs; 240.401µs] or [-0.015%; +0.015%] None None None
credit_card/is_card_number/ throughput [415972669.060op/s; 416094215.378op/s] or [-0.015%; +0.015%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [11.647ms; 11.686ms] or [-0.164%; +0.164%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [8558640.194op/s; 8586708.207op/s] or [-0.164%; +0.164%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [10.804ms; 10.850ms] or [-0.215%; +0.215%] None None None
credit_card/is_card_number/ 378282246310005 throughput [9218356.553op/s; 9258091.104op/s] or [-0.215%; +0.215%] None None None
credit_card/is_card_number/37828224631 execution_time [240.322µs; 240.347µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number/37828224631 throughput [416064954.939op/s; 416107583.366op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number/378282246310005 execution_time [10.469ms; 10.510ms] or [-0.193%; +0.193%] None None None
credit_card/is_card_number/378282246310005 throughput [9516642.680op/s; 9553560.688op/s] or [-0.194%; +0.194%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [10.400ms; 10.440ms] or [-0.188%; +0.188%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [9580822.717op/s; 9616417.362op/s] or [-0.185%; +0.185%] None None None
credit_card/is_card_number/x371413321323331 execution_time [2.191ms; 2.198ms] or [-0.158%; +0.158%] None None None
credit_card/is_card_number/x371413321323331 throughput [45497503.275op/s; 45638558.526op/s] or [-0.155%; +0.155%] None None None
credit_card/is_card_number_no_luhn/ execution_time [240.326µs; 240.349µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ throughput [416062542.437op/s; 416101400.799op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [8.995ms; 9.015ms] or [-0.111%; +0.111%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [11093603.040op/s; 11118004.794op/s] or [-0.110%; +0.110%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [8.161ms; 8.170ms] or [-0.057%; +0.057%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [12240156.816op/s; 12254032.223op/s] or [-0.057%; +0.057%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [240.337µs; 240.362µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [416039767.390op/s; 416082495.721op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [7.819ms; 7.827ms] or [-0.051%; +0.051%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [12776826.224op/s; 12789709.822op/s] or [-0.050%; +0.050%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [10.375ms; 10.412ms] or [-0.177%; +0.177%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [9605923.112op/s; 9639327.080op/s] or [-0.174%; +0.174%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [2.190ms; 2.196ms] or [-0.128%; +0.128%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [45547304.661op/s; 45662910.903op/s] or [-0.127%; +0.127%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 273.311µs 273.760µs ± 0.196µs 273.711µs ± 0.108µs 273.872µs 274.128µs 274.302µs 274.454µs 0.27% 0.909 0.706 0.07% 0.014µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 3643592.341op/s 3652839.120op/s ± 2619.033op/s 3653494.287op/s ± 1446.877op/s 3654568.773op/s 3656329.741op/s 3657200.950op/s 3658841.540op/s 0.15% -0.905 0.697 0.07% 185.194op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 25.745µs 25.786µs ± 0.032µs 25.788µs ± 0.024µs 25.802µs 25.844µs 25.884µs 25.893µs 0.41% 0.876 0.748 0.12% 0.002µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 38620323.241op/s 38780959.411op/s ± 47633.864op/s 38777782.289op/s ± 36650.805op/s 38824249.337op/s 38839780.262op/s 38841634.713op/s 38842000.182op/s 0.17% -0.869 0.726 0.12% 3368.223op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 15.698µs 15.725µs ± 0.030µs 15.731µs ± 0.021µs 15.742µs 15.752µs 15.787µs 16.012µs 1.79% 4.535 40.651 0.19% 0.002µs 1 200
normalization/normalize_name/normalize_name/good throughput 62453291.236op/s 63593457.015op/s ± 119792.740op/s 63569989.241op/s ± 85237.764op/s 63686393.215op/s 63695970.462op/s 63699674.982op/s 63703985.246op/s 0.21% -4.411 39.070 0.19% 8470.626op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [273.733µs; 273.787µs] or [-0.010%; +0.010%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [3652476.147op/s; 3653202.092op/s] or [-0.010%; +0.010%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [25.781µs; 25.790µs] or [-0.017%; +0.017%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [38774357.816op/s; 38787561.007op/s] or [-0.017%; +0.017%] None None None
normalization/normalize_name/normalize_name/good execution_time [15.721µs; 15.729µs] or [-0.026%; +0.026%] None None None
normalization/normalize_name/normalize_name/good throughput [63576854.894op/s; 63610059.137op/s] or [-0.026%; +0.026%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 507.765µs 510.193µs ± 1.182µs 510.157µs ± 0.911µs 511.105µs 511.953µs 512.510µs 514.711µs 0.89% 0.208 -0.157 0.23% 0.084µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1942839.158op/s 1960054.157op/s ± 4538.832op/s 1960181.564op/s ± 3492.642op/s 1963401.392op/s 1967212.697op/s 1968417.793op/s 1969416.582op/s 0.47% -0.195 -0.185 0.23% 320.944op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 471.669µs 472.620µs ± 0.365µs 472.643µs ± 0.265µs 472.910µs 473.215µs 473.344µs 473.422µs 0.16% -0.275 -0.362 0.08% 0.026µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2112282.254op/s 2115863.881op/s ± 1633.903op/s 2115760.377op/s ± 1186.613op/s 2116915.458op/s 2118593.598op/s 2119703.539op/s 2120131.091op/s 0.21% 0.278 -0.359 0.08% 115.534op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 182.583µs 182.847µs ± 0.163µs 182.837µs ± 0.107µs 182.941µs 183.101µs 183.391µs 183.514µs 0.37% 0.743 1.663 0.09% 0.011µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5449165.147op/s 5469069.246op/s ± 4858.546op/s 5469348.370op/s ± 3186.193op/s 5472571.982op/s 5476656.147op/s 5476892.178op/s 5476950.664op/s 0.14% -0.735 1.638 0.09% 343.551op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 44.773µs 44.929µs ± 0.092µs 44.911µs ± 0.056µs 44.981µs 45.098µs 45.191µs 45.280µs 0.82% 0.984 1.003 0.20% 0.007µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 22084812.621op/s 22257297.581op/s ± 45457.712op/s 22266382.114op/s ± 27891.307op/s 22289729.144op/s 22316664.576op/s 22324666.814op/s 22334643.473op/s 0.31% -0.972 0.965 0.20% 3214.346op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 51.219µs 51.832µs ± 0.159µs 51.831µs ± 0.084µs 51.913µs 52.003µs 52.089µs 53.193µs 2.63% 2.682 26.326 0.31% 0.011µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 18799474.411op/s 19293183.846op/s ± 58707.181op/s 19293405.182op/s ± 31195.478op/s 19324555.628op/s 19360135.718op/s 19408664.254op/s 19524013.733op/s 1.20% -2.493 24.615 0.30% 4151.225op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [510.029µs; 510.357µs] or [-0.032%; +0.032%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1959425.118op/s; 1960683.195op/s] or [-0.032%; +0.032%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [472.570µs; 472.671µs] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2115637.438op/s; 2116090.325op/s] or [-0.011%; +0.011%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [182.824µs; 182.869µs] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5468395.898op/s; 5469742.593op/s] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [44.917µs; 44.942µs] or [-0.028%; +0.028%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [22250997.579op/s; 22263597.583op/s] or [-0.028%; +0.028%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [51.810µs; 51.854µs] or [-0.043%; +0.043%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [19285047.596op/s; 19301320.097op/s] or [-0.042%; +0.042%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 138.317µs 139.733µs ± 0.374µs 139.704µs ± 0.191µs 139.920µs 140.390µs 140.941µs 140.988µs 0.92% 0.218 3.071 0.27% 0.026µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [139.681µs; 139.784µs] or [-0.037%; +0.037%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.371µs 3.194µs ± 1.381µs 3.070µs ± 0.033µs 3.097µs 3.160µs 13.724µs 14.920µs 386.04% 7.643 58.726 43.14% 0.098µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [3.002µs; 3.385µs] or [-5.994%; +5.994%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 9.066ms 9.113ms ± 0.025ms 9.112ms ± 0.011ms 9.121ms 9.134ms 9.148ms 9.387ms 3.01% 6.764 73.601 0.27% 0.002ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.109ms; 9.116ms] or [-0.038%; +0.038%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.665µs 2.712µs ± 0.019µs 2.713µs ± 0.011µs 2.723µs 2.745µs 2.759µs 2.768µs 2.02% -0.016 0.303 0.69% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.710µs; 2.715µs] or [-0.096%; +0.096%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 7f4735c 1726747207 vianney/benchmarks-with-dogstatsd
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 36.756µs 37.489µs ± 1.356µs 36.861µs ± 0.051µs 36.964µs 40.405µs 40.444µs 41.420µs 12.37% 1.712 0.983 3.61% 0.096µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [37.301µs; 37.677µs] or [-0.501%; +0.501%] None None None

Baseline

Omitted due to size.

@codecov-commenter
Copy link

codecov-commenter commented Sep 19, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 81.34111% with 64 lines in your changes missing coverage. Please review.

Project coverage is 73.11%. Comparing base (89f48d4) to head (7f4735c).
Report is 1 commits behind head on vianney/main-with-iterations.

Additional details and impacted files 
@@                       Coverage Diff                        @@
##           vianney/main-with-iterations     #641      +/-   ##
================================================================
+ Coverage                         73.05%   73.11%   +0.06%     
================================================================
  Files                               252      252              
  Lines                             36093    36276     +183     
================================================================
+ Hits                              26366    26522     +156     
- Misses                             9727     9754      +27
Components Coverage Δ
crashtracker 20.53% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 93.95% <89.67%> (+3.83%) ⬆️
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 83.08% <ø> (ø)
ddcommon-ffi 69.52% <ø> (ø)
ddtelemetry 59.10% <ø> (ø)
ipc 83.63% <ø> (ø)
profiling 83.57% <ø> (-0.70%) ⬇️
profiling-ffi 77.42% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 38.46% <0.00%> (-1.67%) ⬇️
sidecar-ffi 0.00% <0.00%> (ø)
spawn-worker 50.36% <ø> (ø)
tinybytes 91.66% <ø> (ø)
trace-mini-agent 72.49% <ø> (ø)
trace-normalization 98.25% <ø> (ø)
trace-obfuscation 95.73% <ø> (ø)
trace-protobuf 77.67% <ø> (ø)
trace-utils 92.88% <ø> (ø)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
data-pipeline mini-agent sidecar
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@VianneyRuhlmann @codecov-commenter @ajgajg1134 @bwoebi